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RECOGNIZING COMMANDS WRITTEN ONTO A MEDIUM 

TECHNICAL FIELD 

This invention relates to a machine-executable process 
for recognizing commands written onto a medium, such as paper. 

BACKGROUND 

In the modern workplace a variety of computer resources 
are generally available, such as desk-top or lap-top 
computers. Entering information or commands to a computer is 
typically accomplished by using a keyboard or a mouse 
connected to a computer. 

Despite the availability of computer resources, many 
people still use more traditional methods and mediums to 
record information, such as writing notes and action items on 
paper using a pen or pencil. 

DESCRIPTION OF THE DRAWINGS 

Fig. 1 is a flowchart showing a command recognition 
process • 

Fig. 2A-2B shows exemplary command marks written on 
pieces of paper. 



Docket No.: 10559/582001/P11136 

Fig. 3 is a block diagram of computer hardware on which 
the command recognition process may be implemented. 

DESCRIPTION 

Referring to Fig. 1, a process 10 is shown for 
recognizing commands written on a piece of paper. Process 10 
is basically a three part process. First, a command mark is 
written on a piece of paper 12 using a conventional writing 
implement, such as a pen or pencil. Second, the comjnand mark 
is converted into digital information, either as a digital 
image file 18 or digital stroke data 20, using either a 
scanner 14 or a digital ink capture device 16, respectively. 
Third, the digital information is processed by a recognizer 
program 22 that performs pattern recognition to recognize the 
command mark and convert the recognized command mark into an 
executable command, i.e., a command that may be executed by a 
computer 24. The executable command is either executed 
immediately by computer 24, or, the digital information and 
the converted executable command are stored for later review 
and execution, respectively, on a storage device 26 connected 
to computer 24. 

Performing process 10 enables a writer to write specific 
marks (or strokes) on a piece of paper, which are interpreted 
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as executable commands that are either executed immediately by 
a computer processor, or executed at a later time. Process 10 
simplifies the input, recognition and subsequent management of 
written commands and digital images, using a very common and 
widespread method of storing information, e.g., by writing 
commands and information on a conventional medium, such as a 
piece of paper or cardboard, with a conventional writing 
implement, such as a pen or pencil. Pre-printed material may 
also be processed using process 10 where the pre-printed 
U 10 materials are supplemented with written command marks and then 

O processed by the recognizer program 22. 

W 

s Referring to Fig. 2, exemplary pieces of paper 12a and 

nJ 12b include several written command marks that may be 

recognized and interpreted by process 10. In an embodiment 
according to the invention, command marks are categorized as 
notational, operational and transformational command marks. 
Notational marks typically indicate important phrases, dates 
and keywords. For example, on paper 12a, a star symbol 52 may 
indicate an important item, an underlined phrase 54, 56 and 
58, may indicate a date, a heading or a keyword, respectively. 
Operational marks typically indicate specific operations to be 
performed by a computer 24. For example, on paper 12a, a box 
symbol 60 may indicate an item of information to be included 
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in a task list (a ''to-do'' item) , an envelope symbol 62 may 
indicate an item is to be mailed to another computer user, and 
a filing command symbol 64 may indicate that the digital image 
file represented by paper 12a is to be stored as a file on 
storage device 26. Transformational marks are typically used 
to indicate the transformation (a change) to a specific 
section of text included on a piece of paper, for example, on 
paper 12b, a bold command mark 66 indicates the circled 
section of text 66a should be re-formatted using bold text. 

Referring to Fig. 2B, an exemplary piece of paper 12c, 
includes preprinted text region 90, a special command mark 80 
and secondary command marks 82-84. In an embodiment, special 
command mark 80 must be recognized by recognizer program 22 
before secondary command marks, 82-84 are recognized by 
recognizer program 22, In an embodiment of process 10, 
special command mark 80 is written with a conventional writing 
implement, such as a pen or pencil, onto a conventional pre- 
printed medium, such as piece of paper 12c (or, onto a pre- 
printed piece of cardboard or plastic) , and the conventional 
medium 12c is scanned as a digital image and processed by 
recognizer program 22. In this embodiment, recognizer program 
22 recognizes special command mark 80 before recognizing 
secondary command marks 82-84 in the digital image of paper 
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12c. If special command mark 80 is written on paper 12c, then 
secondary command marks 82-84 are recognized by recognizer 
program 22 and processed as executable commands which may be 
executed immediately or a later time. In an embodiment of 
5 process 10, special command mark 80 is only recognized by 

recognizer program if written on a specific region of paper 
12c, for example, near a corner of paper 12c. 
_ Recognizer program 22 uses a pattern recognition process 

S{ to identify the command marks. The pattern recognition 

2 10 process may be based on a statistical model, a neural network 
model, and/or a Hidden Markov model. 

Recognizer program 22 may apply a machine learning 
ry process to better identify command marks from a particular 

5 writer. The machine learning process may use heuristic 

O 

^ 15 techniques to enhance the pattern recognition accuracy by 
determining common traits and tendencies included in 
previously recognized command marks made by a particular 
writer. 

The recognition of command marks may be intentionally 
20 constrained by recognizing only those command marks made in 
specific regions of paper 30, For example, on paper 12a, 
process 10 may only recognize command marks written to the 
left of a side margin line 68 or above a top margin line 70. 
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The recognized cornmands may be executed immediately by 
computer 24, or may be stored as executable commands in a file 
on storage device 26. Similarly, the digital image file may 
be stored on storage device 26. By storing the command file 
and the digital image file, the writer has the opportunity to 
review both files for accuracy, and make corrections as 
required, before the commands are executed or the text is 
changed by executing a recognized command. 

As discussed previously, process 10 may be performed by 
writing a command mark that is detected by a digital ink 
capturing device 16. Detecting digital ink refers to a 
process of detecting the stroke of a writing implement on the 
surface of a digital ink capture pad. The detected strokes 
are then converted to digital stroke data 20 which may be 
analyzed by recognizer program 22 in a manner similar to the 
scanned digital image 18 described previously. In an 
embodiment of process 10, a piece of paper 12 is placed on top 
of a digital ink capture pad 16 and a writer writes a command 
mark on paper 12. Capture pad 16 detects the strokes made 
through the paper onto the capture pad and converts the stroke 
information into digital stroke data 20. Recognizer program 
22 recognizes the command mark represented by the digital 
stroke data 20 as an executable command and executes the 
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command on computer 24, or stores the executable command in a 
file on storage device 26. 

In an embodiment of process 10, a conventional writing 
implement, such as a pen or pencil, includes a digital ink 
capturing device attached to the writing implement. Command 
marks are then written onto a conventional writing medium, 
such as a piece of paper or cardboard, and the digital ink 
capturing device outputs stroke information which is used by 
recognizer program 22 to recognize command marks. An example 
of a digital ink capturing device that may be attached to a 
writing implement is an optical scanner attached near the tip 
of the writing implement, the optical scanner using a paper's 
surface patterns to detect movements and stroke information 
from the movement of the pen or pencil. Other types of 
digital ink capture devices could be used with process 10 to 
capture digital stroke information based on command marks made 
with a conventional writing implement, such as a pen or 
pencil, onto a conventional writing medium. 

Fig. 3 shows computer 24 on which process 10 may be 
implemented. Computer 24 includes a processor 30, a memory 
32, and a storage medium 26 (see view 36). Storage medium 26 
stores data 38 for digital image files and machine-executable 
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instructions 40 that are executed by processor 30 out of 
memory 32 to perform command recognition process 10. 

Process 10 is not limited to use with the hardware and 
software of Fig. 3. It may find applicability in any 
computing or processing environment. Process 10 may be 
implemented in hardware, software, or a combination of the 
two. Process 10 may be implemented in computer programs 
executing on programmable computers or other machines that 
each include a processor, a storage medium readable by the 
processor (including volatile and non-volatile memory and/or 
storage components), at least one input device, and one or 
more output devices. The input/output device may include 
wireless communications port to send and received signals to 
digital ink capture device. The input/output device may 
include a device that communicates using a wireless protocol, 
such as a Bluetooth protocol. 

Program code may be applied to data entered using an 
input device (e.g., a mouse or keyboard) to perform process 1 
and to generate output information. 

Each such program may be implemented in a high level 
procedural or object-oriented programming language to 
communicate with a computer system. However, the programs ca 
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be implemented in assembly or machine language. The language 
may be a compiled or an interpreted language. 

Each computer program may be stored on a storage medium 
(e.g., a magnetic disk, RAM or ROM) that is readable by a 
general or special purpose programmable computer for 
configuring and operating the computer when the storage medium 
or device is read by the computer to perform process 10. 
Process 10 may also be implemented as a machine-readable 
storage medium, configured with a computer program, where, 
upon execution, instructions in the computer program cause a 
machine to operate in accordance with process 10. 

In an embodiment, process 10 is performed by writing a 
command mark on a conventional medium, such as a piece of 
paper, cardboard or plastic, with the command mark being 
written with a conventional writing implement such as a pen or 
a pencil and the written command mark on the conventional 
medium is then scanned as a digital image which is then 
processed by a recognizer program. 

In an embodiment, process 10 is performed by writing a 
command mark onto a pre-printed medium, such as a pre-printed 
piece of paper or cardboard, with the command mark being 
written with a conventional writing implement such as a pen or 
a pencil, the pre-printed document with the written command 
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mark is then scanned as a digital image which is then 
processed by the recognizer program. 

As used herein, the term ''command mark'' may refers to 
written alphabetic characters and also may refer to glyphs 
that represent non-verbal information (such as a circled 
letter, a star symbol, etc,) 

In an embodiment, the recognizer program uses a 
combination of pattern matching and image processing schemes 
with applied heuristics. The recognizer program may operate 
by performing pattern recognition using either a template- 
based recognition scheme or by employing a Hidden Markov Model 
(HMM) scheme combined with an appropriate feature extraction 
method. The following pseudo-code describes a template-based 
recognition scheme, which may be used to recognize command 
marks by broadly classifying them into ''X-based'' commands and 
''0-based" commands, and then analyzing the recognized command 
marks for further classification. The following pseudo-code 
may also be used to distinguish between a check mark and an X- 
mark, for example: 

1. Segment the scanned image into connected areas. 
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2, For each such segment (corresponding to a letter, a 
word portion, a drawing element, or a command) : 

2a. Is the image of the appropriate size? 
(Discard segments that are too small or too 
large, to avoid conflicts with letters and text) ; 

2b Is the image in the right portion of the page? 
(To enforce a convention that command markes must 
be made in the left half of the image, for 
example) ; 

3 Match the segment to a template: 

3a. Does this segment match an 0-shaped command 
mark? If it does: Draw axes through a centroid of 
a bounding box surrounding the command mark. If 
most of the image data is at approximately the 
same distance from the center of the bounding 
box, this command mark is recognized as an 0- 
shaped command pattern. 
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3b. Or does this segment match an X-shaped 
command? If it does: Draw strips through a 
centroid of a bounding box surrounding the 
command mark. If most of the image data is within 
these strips, then this command mark corresponds 
to an X-shaped command. 

4. Perform more analysis to distinguish the recognized 
command mark from other command marks. For example is the 
command mark an X-shape or an asterisk? Or, is it an 0- 
shape or a cipher? 

In an embodiment, the recognizer program may be used to 
identify text printed on a conventional medium, as described 
previously. The recognizer program may include a scheme to 
recognize the printed text by recognizing at least one common 
characteristic associated with a section of a scanned image. 
For example, recognizing the uniformity of spacings (inter- 
word, inter-line, inter-paragraph) and/or recognizing that 
several letters share the same size or the same font 
attributes (ascenders, descenders, curves), and/or recognizing 
"'neatness", i.e., recognizing at least one margin is 
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substantially aligned with another margin line (for example, 
where margin lines are horizontal, and/or parallel) . 

The recognizer program may apply an image processing 
scheme that enables the recognition of image elements that are 
blurred and/or smeared through resolution changing mechanisms. 
By applying such a scheme in conjunction with Hough 
transforms, printed text of a certain size may be further 
recognized. Similarly, templates based on font attributes 
(curves of ' d' , 'h' , 'p\ or ascenders for 'k', descenders for 
'g', ^j', etc.) can be applied on each recognized text line to 
ascertain uniformity and classify elements as printed. 

The invention is not limited to the specific embodiments 
described above. For example, we mentioned a piece of paper 
as the medium on which command marks are made. However, any 
medium that can be written on and subsequently scanned could 
be used, such as cardboard, metal, plastic, cloth, etc. Also, 
we mentioned using a pen or pencil as the writing implement. 
However, any implement that can be used to make a command mark 
on a suitable medium could be used. 

Other embodiments not described herein are also within 
the scope of the following claims. 
What is claimed is: 



